fix(reports): exact Student-t p-values and a paired comparison section#38
Open
uipreliga wants to merge 6 commits into
Open
fix(reports): exact Student-t p-values and a paired comparison section#38uipreliga wants to merge 6 commits into
uipreliga wants to merge 6 commits into
Conversation
Replace the anti-conservative NormalDist z-approximation with an exact two-tailed Student-t p-value via the regularized incomplete beta (continued fraction, stdlib only) plus Welch-Satterthwaite degrees of freedom. The old approximation understated p at small df and manufactured significance in the small-replicate regime coder_eval runs in (t=2.5, df=4: exact 0.0668 vs approximated 0.0124). Also fixes the docstring's claim of a nonexistent exact fallback, and makes zero-variance-with-different-means return p=0.0 (deterministic difference) instead of 1.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ports Both variants run the same tasks, so an unpaired Welch test carries between-task difficulty variance in its denominator and is systematically underpowered. Add `paired_t_test` (one-sample t on per-task differences, reusing the Phase 1 exact t CDF) and render a `## Paired Comparison` markdown section for any 2-variant experiment. The existing paired score comparison moves out of `## Replicate Statistics` into that section and loses its `replicate_count > 1` gate, so single-replicate A/B experiments — the common case — get it too. Pairing is over per-task mean scores: replicate slots within a task share the task effect and are not independent, so pairing them individually would understate the standard error and manufacture significance (the very failure this change set exists to fix). The task is the unit of analysis, which is also what the section's note reports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Guard the t-test helpers against non-finite input (rubric item 15). student_t_two_tailed_p fails closed to p=1.0; welch_t_test and paired_t_test return None, which renders as "—" rather than a fabricated p-value. - Drop the "more powerful than the pooled Welch test" claim from the Paired Comparison note. Pairing loses degrees of freedom, so it is not unconditionally sharper — the 2-variant snapshot is itself a counter-example (Welch p=0.658 vs paired p=0.766). The note is now descriptive: pairing cancels between-task difficulty. - Stop excluding tasks whose two variants ran unequal replicate counts. That constraint existed to zip replicate slots; pairing per-task means has no such requirement, so the exclusion only discarded real data (and biased the comparison toward fully-completed tasks). - A 2-variant experiment with exactly one common task now says why it has no paired result instead of silently rendering nothing. - Note in-code that the bootstrap CI and the exact-t p-value are separate inference models and can disagree at small task counts. - Warn when the incomplete-beta continued fraction fails to converge rather than returning a partial value silently. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Neither is a ~30-minute guard: the non-finite-input rule needs dataflow analysis rather than the syntactic matching the CExxx rules do, and the bootstrap percentile indexing is latent pre-existing code with no caller that can currently reach it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Out-of-domain values previously surfaced as a bare IndexError from the percentile lookup. Refuse them explicitly instead: clamping into range would quietly return an interval of the wrong width, which is worse than failing. No caller passes either parameter today, so nothing changes for existing report paths. Closes the harness candidate deferred earlier this run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three related report-correctness fixes: - De-duplicate _collect_variant_series. The markdown and HTML reporters each had a copy and they had drifted: the markdown one divides duration_seconds by replicate_count (correct — VariantResult sums it across replicates) while the HTML one used it raw, so HTML showed replicate-inflated durations. Both now call one collect_variant_series in reports_stats. - Replace the paired percentile bootstrap with a Student-t interval. The section printed a bootstrap CI beside an exact-t p-value — two inference models on the same quantity, which could disagree, and the bootstrap was degenerate over the 2-3 task means it now resamples. The CI derives from the same distribution as the p-value, so the two always agree about whether 0 is excluded. New student_t_critical (bisection on the existing t CDF) and paired_t_ci; paired_bootstrap_diff_ci is gone. - Render the paired section in the HTML report too. Both reporters now render one shared reports_stats.paired_comparison, so they cannot disagree about the numbers. Also adds the non-finite harness deferred earlier: it enumerates the module rather than listing helpers by hand, so a helper added later is covered automatically. It immediately caught regularized_incomplete_beta returning NaN for NaN input, now an explicit ValueError. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
uipreliga
requested review from
akshaylive,
bai-uipath and
tmatup
as code owners
July 22, 2026 02:14
|
Claude finished @uipreliga's task in 44s —— View job I'll analyze this and get back to you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
welch_t_testtreated the Welch t-statistic as a z-score (2.0 * NormalDist().cdf(-t)). The normal distribution has lighter tails than Student's t, so this understated p and manufactured significance exactly in the small-replicate regime coder_eval runs in. The in-code comment claimed the approximation "overestimates p slightly" — backwards. Measured: t=2.5 at df=4 is p=0.0668 (not significant); the old code reported 0.0124 (falsely significant). The 2-variant report fixture printedp=0.005where the exact value is 0.106 — understated ~20x.What changed
Exact t-test. Two-tailed Student-t p-value via the regularized incomplete beta (Lentz continued fraction, Numerical Recipes 6.4) plus proper Welch–Satterthwaite degrees of freedom. Pure stdlib —
math.lgamma/log1p— so the module stays zero-dependency; no scipy. Cross-validated againstscipy.stats.ttest_ind(equal_var=False)over 200 randomized group pairs, worst absolute difference 1.4e-13. Also fixes the docstring's claim of an exact fallback that never existed, and makes zero-variance-with-different-means return p=0.0 (a deterministic difference) instead of 1.0.Paired comparison. Both variants run the same tasks, so an unpaired Welch test carries between-task difficulty variance in its denominator and is systematically underpowered. Adds
paired_t_testand a## Paired Comparisonsection for 2-variant experiments. The existing paired block moves out of## Replicate Statisticsand loses itsreplicate_count > 1gate, so single-replicate A/B — the common case — gets it too.Pairing is over per-task mean scores. Replicate slots within a task share the task effect and are not independent; pairing them individually would understate the standard error. The task is the unit of analysis.
One source of truth.
_collect_variant_seriesexisted twice and had drifted: the markdown copy dividesduration_secondsbyreplicate_count(correct —VariantResultsums it across replicates) while the HTML copy used it raw, so the HTML report showed replicate-inflated durations. Both now call onecollect_variant_series. Likewise both reporters render one sharedpaired_comparison(), and the HTML report gains the paired section it never had.Coherent interval. The paired section printed a percentile-bootstrap CI beside an exact-t p-value — two inference models on the same quantity, free to disagree, and degenerate over the 2–3 task means it resamples.
paired_t_cinow derives the interval from the same distribution as the p-value, so the two always agree about whether 0 is excluded.Behavioural deltas (all intended)
[-0.450, +0.200]to[-4.255, +4.005]besidep = 0.766. The bootstrap was resampling two task means and reporting a falsely narrow interval; with 2 tasks you know almost nothing, and the interval now says so.Notes for review
## Paired Comparisonnote is deliberately descriptive and makes no power claim. Pairing trades degrees of freedom for correlation and is not unconditionally sharper — the committed 2-variant snapshot is itself a counter-example (Welch p=0.658, paired p=0.766).student_t_two_tailed_pfails closed to 1.0 (garbage must never read as significant), the list helpers returnNone(rendered—), andregularized_incomplete_betaraises rather than returning NaN.tests/test_reports_stats_nonfinite.pyenumerates the module instead of listing helpers by hand, so a statistic added later is covered automatically. It caught theregularized_incomplete_betaNaN case while being written.Testing
make verify— 3462 passed, 2 skipped, coverage 90.88%. Reference values pinned against scipy without taking a scipy dependency; both markdown snapshots regenerated and reviewed rather than loosened.🤖 Generated with Claude Code